System and method for appliance control via a personal communication or entertainment device

ABSTRACT

A system for use in controlling operating functions of a controllable device includes a hand-held device and an intermediate device in communication with the hand-held device and the controllable device. The hand-held device is adapted to receive a gesture based input and to transmit a signal having data representative of the gesture based input. The intermediate device has programming for translating the data representative of the gesture based input in a signal received from the hand-held device into a command signal to be communicated to the controllable device wherein the command signal has a format appropriate for controlling an operating function of the controllable device that is associated with the gesture based input.

RELATED APPLICATION INFORMATION

This application claims the benefit of U.S. Provisional Application No.61/037,759, filed Mar. 19, 2008, the disclosure of which is incorporatedherein by reference in its entirety.

BACKGROUND

Portable controlling devices, such as for example universal remotecontrols, and the features and functions offered by such devices arewell known in the art. Increasingly, implementations of these devicesincorporate technologies such as color touch screens supportinggraphical user interfaces, wireless home network compatibility, commandrelay stations positioned to control appliances not situated in line ofsight of the controlling device, etc.

Contemporaneously, personal communication, productivity, andentertainment devices such as cellular phones, portable email devices,music players, hand-held games, etc. are also increasingly offeringfeatures such as graphical user interfaces (“GUI”) on color touchscreens, wireless Internet capability, etc. Recently, manufacturers ofsuch platforms have begun to encourage the development of native thirdparty applications which enhance the utility of these devices.

SUMMARY OF THE INVENTION

This invention relates generally to a system and method for enablingcontrol of consumer electronic appliances via a graphical user interfaceimplemented on touch screen equipped personal communication orentertainment devices which include wireless local network capability,such as for example the iPhone® or iPod Touch® products from Apple®.Since such devices do not inherently provide a capability to directlycommunicate commands to the appliances to be controlled, a networkconnected relay station is provided which receives control requests froma software application installed on a personal communication orentertainment device and converts these requests into appliancecompatible infrared (“IR”) or radio frequency (“RF”) commandtransmissions.

In this regard, Apple has recently released a software development kit(“SDK”) for developing and distributing custom third party applicationsfor the iPhone and iPod Touch devices. This SDK provides a full featuredand supported environment for developing software that runs on thedevices, allowing accessory applications that leverage the featuresbuilt into these devices, such as integrated Wi-Fi connectivity,multi-point touch screen interface with gesture recognition, locationservices, audio and video capabilities, 3-axis accelerometer, proximitysensor, etc. Where necessary for a complete understanding of theinvention disclosed herein, exemplary embodiments may be described interms of the Apple products and SDK. However, it will be appreciatedthat other personal communication or entertainment devices which offersimilar feature sets and application support may be equally suitable asplatforms upon which to host comparable features to those describedherein, and accordingly any references to specific Apple products hereinare not intended to be limiting.

In the exemplary embodiments described herein, the universal remotecontrol GUI is implemented as a native application, i.e., designed torun in the computer and operating system environment of a specificpersonal communication or entertainment device or family of devices,collectively referred to hereafter as a “smartphone,” thus enabling thefull capabilities of platform specific features such as for examplemulti-point touch screen interface, motion based gestures, etc., andthereby providing a robust and high quality user experience. This is asopposed to a so-called “web app,” for example one based on HTML pagesserved to an embedded browser such as described for example inco-pending U.S. patent application Ser. No. 12/147,770 “System andMethod for Ubiquitous Appliance Control,” of like assignee andincorporated herein by reference in its entirety. While such web appsare useful when a broad spectrum of different host device types are tobe supported, in general they may not be able to fully leverage deviceunique features in the manner described herein.

A better understanding of the objects, advantages, features, propertiesand relationships of such a portable controlling device system havingcontrol capabilities will be obtained from the following detaileddescription and accompanying drawings which set forth illustrativeembodiments and which are indicative of the various ways in which theprinciples of the system so described may be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various aspects of the portablecontrolling device system having control capabilities describedhereinafter, reference may be had to preferred embodiments shown in theattached drawings in which:

FIG. 1 illustrates an exemplary system in which exemplary personalcommunication device or entertainment devices may be used as controllingdevices;

FIG. 2 illustrates in block diagram form an exemplary command relaydevice of the type used in the exemplary system of FIG. 1;

FIG. 3a through 3d illustrate exemplary touch screen displays of apersonal communication device executing an exemplary remote controlapplication;

FIGS. 4 and 5 illustrate the operation of the exemplary remote controlapplication of FIG. 3 in flow chart form;

FIG. 6 further illustrates the operation of an exemplary remote controlapplication when issuing a request to an exemplary command relay device;and

FIG. 7 illustrates the operation of an exemplary command relay devicewhen processing a request received from an exemplary remote controlapplication.

DETAILED DESCRIPTION

With reference to FIG. 1, according to the teachings set forth ingreater detail below an appliance command relay device 100 is providedwhich receives control request transmissions 120 from a GUI application130 loaded on a smartphone or similar device 102, via a local network.Control requests 120 are transmitted wirelessly from the smartphone 102,for example via a WiFi local network 110. These requests may be receiveddirectly by a command relay device 100 equipped with a compatiblewireless network interface, or may be received by, for example, awireless router 112 and forwarded to a command relay device 100′ via awired connection 114, for example an Ethernet link. In either case, thecommand relay device decodes the received control request transmission,ascertains from it a target appliance, for example TV 108, set top box106, or AV receiver 104, and issues a control command 122 in a formrecognizable by that appliance, for example an IR or RF signal of theformat used by that appliance's remote control device. In this manner,command relay devices 100 and 100′ provide a conduit for the GUIapplication 130 resident in smartphone device 102 to issue operationalcommands to appliances 108, 106, 104 and 108′, 106′ respectively.

While illustrated in the context of a television 108, an STB 106, and anAV receiver 104, it is to be understood that controllable appliances mayinclude, but are not limited to, televisions, VCRs, DVRs, DVD players,cable or satellite converter set-top boxes (“STBs”), amplifiers, CDplayers, game consoles, home lighting, drapery, fans, HVAC systems,thermostats, personal computers, etc. Further, while the exemplaryembodiments presented preferably utilize WiFi-based communication fromsmartphone device 102, it will be appreciated that in other embodimentsdifferent wireless communication protocols such as for exampleBluetooth, Zigbee, Z-wave, etc., may be employed as appropriate.

Turning now to FIG. 2, the architecture of an exemplary command relaydevice is illustrated in block diagram form. For use in commanding thefunctional operations of one or more appliances in response to messagesreceived via a wired or wireless network connection, a command relaydevice 100 may include, as needed for a particular application, aprocessor 200 coupled to a ROM memory 204, a RAM memory 202, anon-volatile read/write memory 206, circuit(s) 208 for transmission ofoperating commands to appliances (e.g., IR and/or RF), a wirelessnetwork transceiver 210 (e.g., WiFi, Bluetooth, etc.) and/or a wirednetwork interface 212 (e.g., Ethernet) for communication with a localnetwork, a means 214 to provide feedback to the user (e.g., one or moreLEDs, LCD display, speaker, and/or the like), and a power source 216.

As will be understood by those skilled in the art, some or all of thememories 202, 204, 206 may include executable instructions(collectively, the program memory) that are intended to be executed bythe processor 200 to control the operation of the command relay device100, as well as data that serves to define appliance control protocolsand command values to the operational software (the appliance codedata). In this manner, the processor 200 may be programmed to controlthe various electronic components within the command relay device 100and process the input and output data thereof, for example, to receiveand transmit data via network interfaces 210 and/or 212, to act uponcommands and requests embodied in such data, to cause the transmissionof appliance command signals via transmission circuits(s) 208 toappliances to be controlled, to control visual feedback device(s) 214,etc. While the memory 204 is illustrated and described as a ROM memory,memory 204 may also be comprised of any type of readable media, such asROM, FLASH, EEPROM, or the like. Preferably, the memories 204 and 206are non-volatile or battery-backed such that data is not required to bereloaded after power interruptions. In addition, the memories 202, 204,and 206 may take the form of a chip, a hard disk, a magnetic disk, anoptical disk, and/or the like. Still further, it will be appreciatedthat some or all of the illustrated memory devices may be physicallyincorporated within the same IC chip as the microprocessor 200 (a socalled “microcontroller”) and, as such, they are shown separately inFIG. 2 only for the sake of clarity.

To cause the command relay device 100 to perform an action, commandrelay device 100 is adapted to be responsive to events, such as areceived signal from network interface port 210 or 212. In response toan event, appropriate instructions within the program memory (hereafterthe “operating program”) may be executed. For example, receipt of acontrol request message from smartphone device 102 may result in theretrieval from the appliance code data the command value and controlprotocol appropriate for an intended target device and a resultingtransmission of the requested command to the intended target appliance,e.g., the STB 106, in a format recognizable by the intended targetappliance.

For selecting a set of appliance code data to be associated with anappliance to be controlled, data may be provided to the command relaydevice 100 that serves to identify an intended target appliance by itstype and command protocol. Such data, provided as part of or separatelyfrom a command request, may allow the command relay device 100 toidentify the appropriate appliance code data elements within apreprogrammed library of appliance code data, to be used to transmitrecognizable commands in a format appropriate for such identifiedappliances. Alternatively, either in place of or in addition to apre-stored library, appliance code data may be downloaded into commandrelay device 100 via a network interface(s) 210, 212 either during aninitialization phase or on an as required basis.

Operation of the Remote Control Application.

Turning now to FIG. 3, a third party remote control application may beprovided for purchase and download into a smartphone, for examplethrough Apple's iPhone App Store—apple.com/iphone/appstore. Oncedownloaded and installed, the availability of an exemplary remotecontrol application may be represented on a smartphone home screen 300(such as shown in FIG. 3a ) by the presence of an icon 302. Activationof icon 302, i.e., execution of the remote control application, mayresult in a GUI screen 304 such as illustrated in FIG. 3b . This GUI maycomprise selection icons 306 representative of the appliances that theapplication and its associated command relay station are currentlyconfigured to control, together with configuration icon 308 activatableto add or reconfigure controlled devices and icon 310 activatable toreturn to the smartphone home screen 300. Activation of a device icon306, for example the icon labeled “TV,” may result in the presentationof a set of appliance control icons 320 such as illustrated in FIG. 3c .The specific set of appliance control icons presented may be dependenton the device type selected. In the event more device control functionsare available than can reasonably be presented in a single display page,additional display pages may be provided, accessible via the smartphonedevice's usual page scrolling functionality (for example, a horizontalswipe of a finger in the case of an Apple iPhone.)

The functionality of various icons illustrated in FIGS. 3b through 3dwill now be described in greater detail in conjunction with theflowcharts of FIGS. 4 and 5, which are representative of the processingperformed by an exemplary remote control application, commencing at step400 when an event such as a screen touch, accelerometer output,proximity sensor, timer expiry, etc., is detected by the smartphoneoperating system and presented to the exemplary remote controlapplication for processing. It will be appreciated that in these andother flowcharts presented herein by way of illustration, stepsassociated with common housekeeping functionality as is well known inthe art, for example low battery detection, display timeouts, errorconditions, etc., may be omitted for clarity. First, at step 402 it isdetermined if the event is an incoming phone call. If so, at step 404the current status of the remote control application is saved in orderto allow the application to return to the same operational screen whenre-launched, for example at the conclusion of a phone conversation.Next, at step 406 it is determined if the automatic mute/pause feature(a user option in the exemplary embodiment) is enabled. If so, at step408 a “mute” command request targeted to the volume control applianceand/or a “pause” command request targeted to the media playbackappliance may be issued to the command relay station, after which atstep 440 the exemplary remote control application relinquishessmartphone device control to the phone mode.

If the event is not an incoming call, at step 410 it is determined ifthe event is the result of a user interaction with the device such abutton tap, finger swipe, accelerometer based input, proximity sensorinput, etc. If so, it is next determined at step 412 if the currentstatus of the remote application is gesture mode. If so, processingcontinues at step 500 in FIG. 5, as will described hereafter. If thestatus of the remote control application is not gesture mode, it is nextdetermined at step 414 if the event was activation of a button icon. Ifso, at step 416 it is determined if the activated icon was a “close”icon 310 or 326. If so the exemplary remote control application closesand relinquishes smartphone device control to the default phone mode ofoperation. If a “close” icon was not activated, it is next determined atstep 418 if the activated icon was one of the appliance selection icons306. If so, at step 436 the current appliance to be controlled is set asindicated by the particular appliance selection icon activated, and thefirst page of command icons applicable to the selected appliance isretrieved and displayed. By way of example, if the selected appliance isa TV, the displayed page may be as illustrated at 320 in FIG. 3c . Itwill be appreciated that the contents of this first, initiallydisplayed, page may vary with appliance type—for example while a TVappliance page may be as illustrated, a DVD appliance first page maycomprise transport controls (play, pause, fast forward, etc.), a cableSTB first page may comprise program guide navigation and selectioncontrols, etc.

If the event is not an appliance selection, at step 420 it is nextdetermined if the activated icon is a request to perform an operationinternal to the remote control application, for example, via activationsof icons 322 for scrolling to a control page for a different type ofappliance, icon 324 for returning to the remote control “home” page (forexample, appliance selection page 304), icon 326 for exiting the remotecontrol application, etc. If so, at step 422 the requisite action isperformed.

If the activated icon is not to result in the performance of an internaloperation, the icon activation may represent an appliance commandoperation, for example a request to issue a TV “power” command initiatedvia icon 328. If so, at step 432 the requested command function isdetermined and at step 434 an appropriate command request is issued tocommand relay device 100. The requested command function may also beinternally stored by the remote control application for futurereference.

If the event is not an icon activation, at step 428 it is nextdetermined if the event comprises a shake, i.e. rapid movement along twoor more axes as reported by the accelerometer hardware of the smartphonedevice. If so, in an exemplary embodiment this action by the user isinterpreted as a request to repeat the last appliance command issued. Inthis case, at step 430 the command previously stored at step 434 aboveis retrieved and a request for that command issued to command relaydevice 100.

If not a shake, at step 436 it is next determined if the event comprisesa touch screen gesture by the smartphone user. If so, this event isprocessed as will be described hereafter in conjunction with FIG. 5. Ifnot, no recognizable event has occurred, and the remote controlapplication terminates current processing at step 438.

If the detected event is not an incoming call or a user interaction, atstep 424 it is determined if the event is a timer expiry. If not, norecognizable event has occurred, and the remote control applicationterminates current processing at step 438. If so, at step 426 anygesture mode status and screen overlay display(s) are cleared. By way ofexplanation, during certain gesture activities a semi-transparent overlymay be displayed as described in greater detail hereafter in conjunctionwith FIG. 5. While this display is present, a time-out may beestablished in order to remove the display overlay after a predeterminedperiod of inactivity.

Turning now to FIG. 5, if at step 436 it was determined that the userinput constituted a gesture (i.e., a touch screen interaction of aparticular type, as detectable by the smartphone vendor-suppliedoperating system), at step 506 it is determined if the gesture was asimple scroll request, for example a horizontal swipe which may beinterpreted as a request to scroll to the next page of appliance commandicons in instances where multiple pages of command functions exist for aparticular appliance type. If so, the request is processed at step 518and the remote control application thereafter terminates currentprocessing at step 516. If not a scroll request, at step 508 it is nextdetermined if the gesture comprised a two finger double tap. If so, inthe exemplary embodiment this may be interpreted as a user request toenter into a volume and channel control mode. Additionally, in theexemplary embodiment, repeat of a mode setting gesture while already inthat mode may be interpreted as a request to terminate that mode ofoperation. Accordingly, at step 520 it is first determined if the remotecontrol application is already operating in the volume and channelcontrol mode. If so, processing continues at step 426 to clear thatmode. If not, at step 522 the gesture mode status is set to “Vol/Chancontrol,” at step 524 a semi-transparent overlay, for example asillustrated at 330 of FIG. 3d , is displayed to indicate the mode ofoperation, and at step 540 a gesture mode timer is started (orrestarted, if the remote control application was already operating inone of the gesture modes), after which processing of the current eventcompletes at step 516.

If not a two finger double tap, at step 510 it is determined if theevent comprised a three finger double tap. If so, in the exemplaryembodiment this may be interpreted as a user request to enter into atransport control mode, i.e., offering control of the “play”, “pause”,“fast forward”, “rewind”, etc. functions of a media playback appliance,and at steps 526 through 530 the request is processed in a similarmanner to that previously described, resulting in the display of atransport function semi-transparent overlay (not illustrated).

If not a three finger double tap, at step 512 it is determined if theevent comprised a pinch close gesture (two digits, typically a fingerand thumb, placed in contact with the touch sensitive surface and movedin opposing directions towards one another). If so, in the exemplaryembodiment this may be interpreted as a user request to enter into amenu navigation mode, i.e., offering control of the “select”, “up”,“down”, “left”, “right,” etc. functions of an appliance, and at steps532 through 536 the request is processed in a similar manner to thatpreviously described, resulting in the display of a menu navigationfunction semi-transparent overlay (not illustrated).

If not a pinch close gesture, at step 514 it is next determined if theevent comprised a two finger vertical swipe. If so, in the exemplaryembodiment this may be interpreted as a user request to issue a mutecommand to the current audio rendering appliance, and at step 538 therequested appliance command is set for issuance to command relay station100. If not a two finger vertical swipe, no recognizable event hasoccurred, and the remote control application terminates currentprocessing at step 516.

When the remote control application status indicates that it isoperating in the gesture mode (i.e., a mode previously set at one ofsteps 522, 528 or 534 and not dismissed or timed out) receipt of a userinteraction event at step 412 causes processing to continue at step 500where it is determined if the event is a horizontal or vertical fingerswipe. If so, the appropriate appliance command is selected at step 504and a request issued to command relay device 100 via step 434. Selectionof appliance type and command is based on the current gesture status,for example an upward swipe may indicate “channel up” if involume/channel control mode, “stop” if in transport control mode, “menuup” if in menu navigation mode, etc. If not a finger swipe, at step 501it is next determined if the event is a finger tap in the centralportion of the screen. If so, processing continues at step 504 with theselection of an appropriate command (for example “mute”, “pause”, or“select” dependent once again on the current gesture status.) If not, atstep 502 it is determined if the event is activation of dismiss icon332, indicating that the user wishes to exit gesture mode. If soprocessing continues at step 426 where the gesture mode status andscreen overlay display are cleared. If not processing continues at step508 as previously described.

It will be appreciated that additional or alternative gestures and/orcommand functions may be implemented in this or other embodimentsdependent on the gesture recognition capabilities of specific smartphoneor entertainment device platforms, including those that may comprise twoor more types of gestures performed simultaneously or in concert withone another in order to accomplish more complex command functions. Byway of example, in one embodiment of the present invention a rampingfast forward or reverse function may be enabled via a single buttonpress and hold input on the touch screen, during which accelerometerbased input (e.g., tilting the smartphone to the right or left)gradually increases or decreases the rate at which media isfast-forwarded or reversed. Furthermore, in some embodiments theassignment of particular gestures to specific command functions mayconfigurable at user option. Accordingly, it is to be understood thatthe gesture functionality presented in the foregoing paragraphs is byway of example only and not intended to be limiting.

Request Protocol Between Remote Control Application and Command RelayDevice.

In an exemplary embodiment command request and response data packets ofthe format described below may be exchanged between a remote controlapplication and a command relay device via a wireless 110 and/or wired114 local area network.

A command request from remote control application to a command relaydevice comprises the following data fields (values in parentheses aftereach field name represent the size of that field in bytes, i.e., thenumber of digits or characters):

-   -   SystemID(4), SpecRevision(4), RcdLength(2), RequestType(1),        ApplianceType(2), ApplianceNum(4), CommandCode(2), KeyFlag(1)

Where:

SystemID may be a unique system identification number established forexample during an initial discovery and pairing process between a remotecontrol application and a command relay device with which it is to beassociated.

SpecRevision may be a version/revision indicator of the interfacespecification to which this data packet format conforms, for example toensure compatibility between devices or for determining data packetfield locations and values.

RcdLength may be the number of data characters following this field(i.e., excluding itself)—“10” in the case of this exemplary commandrequest packet.

RequestType may be the type of request—In the example presented, “01”may represent a request to commence transmitting an appliance command,“02” may represent a request to continue transmitting an appliancecommand, and “00” may represent a request to cease transmitting anappliance command. See the narrative below regarding the KeyFlag fieldfor further insight regarding this aspect of the request protocolbetween the exemplary remote control application and the command relaydevice.

ApplianceType may be the type of appliance to be commanded, for exampleaccording to the following table:

TABLE 1 Exemplary appliance types. Appliance Type Device Name 00 TV 01Cable STB 02 Video Accessory 03 Satellite STB 04 VCR 05 Laser Disk 06DVD 07 Tuner/AV Receiver 08 Amplifier 09 CD 10 Home Control 11-31Reserved

ApplianceNum may be a four digit number which identifies a set ofcommand data and transmission format information, i.e., the appliancecode data, required to control a particular appliance or family ofappliances. This may for example represent a set up code or applianceidentifier as is well known in the universal remote control art, see forexample “ATLAS OCAP 5-Device Remote Control” user manual, Revision 3.0,by Universal Electronics Inc., incorporated herein by reference in itsentirety.

CommandCode may be a two digit identifier of the specific commandrequested, for example according to the following table:

TABLE 2 Exemplary appliance command functions Standard Command #Function 01 POWER TOGGLE 02 POWER ON 03 POWER OFF 04 CHANNEL UP 05CHANNEL DOWN 06 VOLUME UP 07 VOLUME DOWN 08 MUTE 09 DIGIT 1 10 DIGIT 211 DIGIT 3 12 DIGIT 4 13 DIGIT 5 14 DIGIT 6 15 DIGIT 7 16 DIGIT 8 17DIGIT 9 18 DIGIT 0 19 CHANNEL ENTER 20 +100 21 LAST CHANNEL 22 INPUT 23EXIT 25 STOP 26 PAUSE 27 REWIND 28 FAST FORWARD 29 RECORD 30 SKIP FWD 31SKIP REVERSE 32 LIVE 33 SOURCE MENU 34 DEVICE MENU 35 GUIDE 36 EXIT 37BACK 38 CURSOR UP 39 CURSOR DOWN 40 CURSOR LEFT 41 CURSOR RIGHT 42 MENUSELECT 43 PAGE UP 44 PAGE DOWN 45 FAVORITE 46 DISPLAY 47-99 Reserved

KeyFlag may be a directive which may control the style of commandtransmission to the appliance. By way of example, in an exemplaryembodiment KeyFlag=“1” may request that the appliance command be issuedonce by the command relay device, while KeyFlag=“0” may request that theappliance command be continuously transmitted by the command relaydevice until an explicit “cease transmitting” request is received—used,for example, in the case of ramping functions such as volume controlwhere a command transmission should be continuously repeated for as longas a user keeps a key depressed, a channel surfing function in whichchannels are to be incrementally viewed, etc.

A response from a command relay device to a remote control applicationmay comprise the following fields:

-   -   SystemID(4), SpecRevision(4), RcdLength(2), RequestType(1),        ApplianceType(2), ApplianceNum(4), CommandCode(2), Status(2)

Wherein the first seven fields may be as described above and maycomprise an echo of the received request by way of confirmation, and thefinal Status field may comprise one of the values shown in Table 3below, representing the completion status of the command request.

TABLE 3 Exemplary status response values Status value Message 00Completed without error 01 Invalid appliance type 02 Invalid appliancenumber 03 Invalid request type 04 Invalid command code 05 No commanddata available 06 Data Packet Format Error 07 Continuous commandtransmission aborted by timeout 08-99 Reserved

Turning now to FIG. 6, the steps performed by an exemplary embodiment ofa remote control application in issuing a command request in accordancewith the exemplary data protocols above will be described in furtherdetail, i.e., a series of steps which may correspond to the process ofstep 434 of FIG. 4. First at step 602 exemplary packet header data isinitialized, comprising for example appropriate SystemID, SpecRevision,and RcdLength fields together with a RequestType field value of “01”(commence transmitting a command). Next, at steps 604 and 606, exemplaryApplianceType, ApplianceNum, and CommandCode fields are populatedaccording to the indicated appliance and function request. At step 608the requested command is also saved locally for possible future use inprocessing a possible “repeat last command” request as previouslydescribed in conjunction with FIG. 4. At step 610 it is determined ifthe requested is for a ramping function. By way of example, when anexemplary remote control application is operating in the volume/channelgesture input mode illustrated in FIG. 3d , a left to right finger swipeterminating with the finger removed from contact with the touchsensitive surface may be interpreted as a request to issue a singleinstance of a “volume up” command to the designated appliance, i.e., anon-ramping function, whereas a left to right finger swipe terminatingwith the finger remaining in stationary contact with the touch sensitivesurface may be interpreted as a request to continuously issue “volumeup” commands to the designated appliance until such time as the fingeris lifted, i.e., a ramping function. At steps 612 and 614 the KeyFlagfield value is set according to this determination. At step 616, thecompleted request data packet is forwarded to an exemplary command relaydevice via wireless local area network 110. As will be appreciated, thedata packet may be encapsulated for transmission purposes within furtherheader fields, control parameters, error detection and/or correctiondata, etc., as appropriate for the wireless protocol being employed, forexample WiFi, Bluetooth, Zigbee, Zensys, etc., all as well known in theart.

After transmission of the request data packet, at steps 618, 620 and 622the remote control application waits for a response from the exemplarycommand relay device. If the received response is negative, i.e., anyvalue other than “00” in the example presented, or if no response isreceived within a predetermined amount of time, an error conditionexists and is processed at step 652. The exact actions taken may dependupon both the type of error and the activity currently being requested.By way of example without limitation, a timeout or receipt of a statuscode such as 06 “data packet format error” may result in re-transmissionof the request packet; receipt of status code 05 “no command dataavailable” may result in graying out of particular button icons incertain display pages; receipt of status code 02 “invalid appliancenumber” may result in a message to the user suggesting re-initializationof appliance settings within the smartphone device and/or downloading ofadditional code data to command relay station 100; while other errorstatus codes may result in display of error messages requesting possibleaction by the user such as reconfiguring or reinitializing devices, etc.

If a good status response is received, at step 624 the remote controlapplication next determines if the KeyFlag value of the acknowledgedrequest was “01.” If so, the function requested was the transmission ofa single instance of the appliance command represented by CommandCode,and processing is complete. If not, the request is for a continuoustransmission of the appliance command for as long as the key or otheruser input remains activated. In that case, at step 626 a predeterminedkey timeout is first initiated as a precaution in the event that thecontinued key activation is not the result of a user action but ratheran error condition—for example a smartphone device has become lodgedbetween couch cushions, an object has been inadvertently placed on orhas fallen onto the touch screen surface, etc. Should this timeoutexpire, the remote control application will subsequently act as if thekey had been released regardless of its activated state. Then, at step628 it is determined if the key or input is in fact still activated andif so, provided the above referenced key timeout has not expired (step630), at steps 632 and 634 a further request packet is transmitted tothe exemplary command relay device, with RequestType set to “02—continuetransmitting.” As will be detailed hereafter, in certain embodiments anexemplary command relay device which is transmitting a commandcontinuously may require periodic receipt of such “continuetransmitting” requests from the remote control application in order tosustain the transmission, this representing a safety feature to preventrunaway of an appliance adjustment function in the event thatcommunication between the remote control application and the commandrelay device is interrupted prior the issuance of a “cease transmitting”request.

Upon completion of transmission of the packet, the remote applicationonce again waits for a response from the exemplary command relay device.If a bad response or timeout occurs, the error condition is processed atstep 652 as previously described. If a good response is received, thesequence is repeated starting at 628 until such time as the key or inputis no longer activated or the key timer times out. Upon occurrence ofeither of these events, at steps 642 and 644 a request packet istransmitted to the exemplary command relay device, with RequestType setto “00—cease transmitting”, the ensuing response is evaluated at steps646, 648 and 650, and processing is complete.

Operation of the Command Relay Device.

Turning now to FIG. 7, the steps performed by an exemplary embodiment ofa command relay device in processing a command request in accordancewith the exemplary data protocols above will be described in furtherdetail. Upon receipt of a command request data packet in conformancewith the protocol described previously, at initial step 702 the packetheader data is examined. For example, the exemplary SystemID andSpecRevision fields may be inspected to determine if they match expectedvalues. An exemplary RcdLength field may also be evaluated at this stepto determine if a complete data packet has been received. If any ofthese tests fail, a data packet that is broken and/or of unknown formathas been received, and accordingly at step 704 a status “data packetformat error” is set whereafter the balance of the received packetfields are ignored, a response containing that status is returned to theoriginating remote control application at step 750, and processing iscomplete.

Next, at step 706 an exemplary RequestType field is examined todetermine if the received data packet is a “00—cease transmitting”request. If so, at step 732 and 733 a status “good completion” is setand any timeout timer is cleared, whereafter any ongoing appliancecommand transmission is cancelled at step 746, a response containing thegood status is returned to the originating remote control application atstep 750, and processing is complete.

If the exemplary RequestType field is not a cease transmitting request,at step 708 it is determined if the received RequestType field is a“02—continue transmitting” request. If so, at step 709 a status “goodcompletion” is set, whereafter at step 748 an exemplary transmissiontimeout is reset, a response containing the good status is returned tothe originating remote control application at step 750, and processingis complete.

If the exemplary RequestType field is not a continue transmittingrequest, at step 710 it is determined if the received RequestType fieldis a “01—transmit command” request. If not, other type of requests maybe processed at step 712. By way of example these may include devicediscovery and pairing transactions, initial and setup and configurationcommunications, etc. without limitation, as required by particularembodiments.

If the request is for a command transmission, at step 714 the exemplaryApplianceType and ApplianceNum fields are evaluated. If the requestedappliance type and/or number are invalid, i.e., do not correspond to anidentifiable portion of the appliance code data currently stored in thememories 202, 204, 206 of command relay device 100, an appropriate errorstatus is set at step 716 and processing continues at step 750 aspreviously described.

If the requested appliance type and number are valid, at step 718 theseare registered as the appliance to be controlled, for example byestablishing a pointer to the appropriate portion of stored appliancecode data. At step 720, the contents of the exemplary CommandCode fieldare then examined. If there is not a recognizable command code, an errorstatus is set at step 722 and processing continues at step 750 aspreviously described. If a valid command code is present, at step 724and 726 the entry within the stored appliance code data whichcorresponds to the request appliance type, number, and command isexamined. If no data is available, an error status is set at step 728and processing continues at step 750 as previously described. By way ofexplanation, not all appliances necessarily support every functiondefined in the universe of possible appliance commands. By way ofspecific example, a particular appliance may not support a discrete“power on” function even though in the exemplary embodiment illustratedin Table 2 this is a permissible command (“02”), in which case theappliance code data entry corresponding to that appliancetype/number/command may be null, resulting in the “no command dataavailable” status of step 728.

If valid data is present, then at steps 730 and 732 the software driverassociated with IR or RF transmitter 208 is initialized using the storedformatting information corresponding to the selected appliance type andnumber. Such formatting information may include for example a carrierfrequency, duty cycle, encoding schema, frame length, etc., as wellknown in the art. Thereafter, at steps 738 and 740 the command data isretrieved and transmission commenced. Once command transmission isinitiated, the response status is set to “good” at step 742.

The exemplary KeyFlag field is then examined at step 744. If the contentis non-zero, processing continues at step 746 and command transmissionis stopped after completion of a single instance. In this regard, itwill be appreciated that a single instance of a command may comprisetransmission of more than one frame of data, depending on the applianceprotocol is use. By way of specific example, the error detectionmechanism of a certain manufacturer's IR command protocol may requirethat a frame of command data be repeated multiple times for comparisonpurposes before it will acted upon. Accordingly it is to be understoodthat at step 746 transmission will only cease after completion of allrequirements of the particular appliance command protocol currently inuse. Once complete, processing continues at step 750 by issuing acompletion response to the requesting smartphone device, after whichprocessing is complete.

If the content of the exemplary KeyFlag field is zero, the requestingsmartphone device wishes to initiate a ramping function, and accordinglyat step 748 an exemplary transmit timeout is initiated, whereafter acompletion status is returned to the requesting smartphone device andprocessing completes with command transmission ongoing.

As mentioned previously, the purpose of the timeout initiated at step748 is to prevent a runaway command situation in the event thatcommunication between smartphone 102 and command relay device 100 isinterrupted and a timely “Cease transmitting” request is not received bythe command relay device. Accordingly, if this timeout expires prior toreceipt of any “Continue transmitting” or “Cease transmitting” requestfrom the smartphone device, at step 734 a response status “Aborted bytimeout” is set, after which any ongoing command transmission isterminated and that status is communicated to the smartphone device.

Initialization of the System.

A newly-installed remote control application when run for the first timeon a smartphone device may initiate a discovery process to locate andpair with a command relay device 100 attached to the local wirelessnetwork. In certain embodiments, upon discovering a command relay devicethe remote control application will instruct that command relay deviceto visually or audibly signal the user, for example using statusindicator 214, and then prompt the user for a confirmation input, forexample via the touch screen. If no user confirmation is received withina predetermined period of time the remote control application may resumethe discovery process after flagging the current command relay device soas to not identify it again during the current run instance. In thismanner allowance may be made for the presence of more than one commandrelay device on the same local network.

Once paired with a command relay device, in certain embodiments theremote control application may immediately enter an “add device” modesoliciting user configuration of the appliance types, brands and/ormodels to be controlled. Alternatively, the remote control applicationmay simply display a blank home screen (similar to screen 304 sansappliance selection icons 306) awaiting manual user initiation of an adddevice mode, for example via activation of icon 308. In either event,multiple methodologies for identification of appliances to becontrolled, whether performed locally or by interaction with a Web-basedservice are well known in the art, by way of example without limitation,as may be found described in U.S. Pat. No. 4,959,810 “Universal RemoteControl Device” or U.S. Pat. No. 7,218,243 “System and Method forAutomatically Setting Up a Universal Remote Control,” both of likeassignee and incorporated herein by reference in their entirety. In someembodiments, appliance identification may be performed by interactingwith a web-based service which may convert user-supplied appliance brandand model information into appliance type and number values suitable foruse by the smartphone application. In some instances, supplementalappliance code data not already present in the memories of command relaydevice 100 may be downloaded into the device via network connections 114or 120. In certain embodiments this may occur based on a web service'sprior knowledge of the current library of appliance code data in commandrelay device 100, while in other embodiments this may be triggered, forexample, by a smartphone application upon receipt of an “Invalidappliance number” or “Invalid appliance type” status response fromcommand relay device 100.

In some embodiments, various further options may be configurable withinthe remote control application either during the initialization processor separately thereafter, for example the optional pause and/or mutefeatures illustrated at steps 406 and 408 of FIG. 4 may be enabled ordisabled, individually or jointly. Additionally the assignment ofappliance type, command functions and/or remote control applicationlocal functions to specific gestures may be user configurable. It willbe appreciated that in certain embodiments, the device type to beassociated with a specific gesture may be globally assignable, i.e.,effective without regard to the currently selected device of the remotecontrol application (for example, the volume control functions may beglobally assigned to TV 104, while channel up/down functions may beglobally assigned to set top box 106), while in other embodiments thedevice type may be that currently selected within the remote controlapplication. In this connection, functions represented by conventionalbutton icons which have also been assigned a gesture, for example thechannel up icon 340 and channel down icon 342 of FIG. 3c may be visuallyidentified as such by the addition of a badge or tag 344 appended to orplaced in the proximity of that function icon.

While various concepts have been described in detail, it will beappreciated by those skilled in the art that various modifications andalternatives to those concepts could be developed in light of theoverall teachings of the disclosure. For example, it will be appreciatedthat alternative and/or additional gestures may be utilized in variousembodiments of the invention, and that the GUI displays andfunctionality presented herein are illustrative only. For example, incertain embodiments icons 322, 324 and 326 may be utilized to accessuser-assignable favorite channels rather than the internal functionsdescribed herein. Further, certain embodiments of the remote controlapplication may take advantage of additional capabilities of thesmartphone platform in which they are resident. For example a remotecontrol application resident in a smartphone platform capable of spatialawareness (e.g., via GPS, wireless network detection, or other locationbased technology) may automatically select appropriate appliances to becontrolled as a user moves from room to room and/or to automaticallyalter which appliance is assigned to various gestures. Additionally,while combination gestures (for example a finger swipe followed by atap, tilt, shake, etc.) may be presented in the exemplary embodimentsillustrated herein as being fully interpreted within a smartphoneplatform, it will be appreciated that in alternative embodiments each ofthe individual gesture component actions may be separately reported to acommand relay device and interpreted thereat.

While described in the context of functional modules and illustratedusing block diagram and flow chart formats, it is to be understood that,unless otherwise stated to the contrary, one or more of the describedfunctions and/or features may be integrated in a single physical deviceand/or a software module, or one or more functions and/or features maybe implemented in separate physical devices or software modules. It willalso be appreciated that a detailed discussion of the actualimplementation of each module is not necessary for an enablingunderstanding of the invention. Rather, the actual implementation ofsuch modules would be well within the routine skill of an engineer,given the disclosure herein of the attributes, functionality, andinter-relationship of the various functional modules in the system.Therefore, a person skilled in the art, applying ordinary skill, will beable to practice the invention set forth in the claims without undueexperimentation. It will be additionally appreciated that the particularconcepts disclosed are meant to be illustrative only and not limiting asto the scope of the invention which is to be given the full breadth ofthe appended claims and any equivalents thereof.

All patents cited within this document are hereby incorporated byreference in their entirety.

What is claimed is:
 1. A system for use in controlling operatingfunctions of a target controllable device, comprising: a hand-helddevice adapted to receive a gesture based input and having programmingfor transmitting a signal having data representative of the gesturebased input, data representative of device type for the targetcontrollable appliance, and data representative of a command code dataset identifier; and an intermediate device in communication with thehand-held device and the controllable device, the intermediate devicehaving programming which causes the intermediate device to respond to areceipt of the signal from the hand-held device by using the datarepresentative of the device type for the target controllable applianceand the data representative of the command code data set identifier toselect a command code data set from a library of command code data setsstored in a memory of the intermediate device, using the datarepresentative of the gesture based input to identify at least onecommand code in the selected command code data set, and using theidentified at least one command code in the identified command code dataset to create a command signal to be communicated to the targetcontrollable device wherein the command signal has a format appropriatefor controlling an operating function of the target controllable devicethat is associated with the gesture based input.
 2. The system asrecited in claim 1, wherein the library of command code data sets ispre-stored in the memory.
 3. The system as recited in claim 1, whereinat least a portion of the library of command code data sets isdownloaded from a network server device and then stored in the memory.4. The system as recited in claim 1, wherein the hand-held devicecomprises a touch screen interface and the gesture based input comprisesa sensed interaction with the touch screen interface.
 5. The system asrecited in claim 4, wherein the gesture based input comprises a sensedswiping motion on the touch screen interface.
 6. The system as recitedin claim 4, wherein the gesture based input comprises a sensed, plural,simultaneous touching on the touch screen interface.
 7. The system asrecited in claim 4, wherein the gesture based input comprises a sensed,simultaneous swiping motion and touching on the touch screen interface.8. The system as recited in claim 1, wherein the hand-held devicecomprises an accelerometer and the gesture based input comprises amovement of the hand-held device sensed via the accelerometer.
 9. Thesystem as recited in claim 1, wherein the programming of theintermediate device functions to transmit to the hand-held device astatus signal having data indicative of a status of the intermediatedevice resulting from receipt of a signal transmitted from the hand-helddevice.
 10. The system as recited in claim 1, wherein the datarepresentative of the gesture based input comprises data indicative ofan operating function type.
 11. The system as recited in claim 1,wherein the data representative of the gesture based input comprisesdata indicative of an instruction for the intermediate device torepeatedly communicate a command signal to the target controllabledevice.
 12. The system as recited in claim 11, wherein the instructioncauses the intermediate device to repeatedly communicate a volumecontrol command signal to the target controllable device.
 13. The systemas recited in claim 11, wherein the instruction causes the intermediatedevice to repeatedly communicate a channel tuning control command signalto the target controllable device.
 14. The system as recited in claim 1,wherein the data representative of the gesture based input comprisesdata indicative of a request type for the intermediate device.
 15. Thesystem as recited in claim 1, wherein the intermediate devicecommunicates with the target controllable device via a wireless signaltransmission.
 16. The system as recited in claim 1, wherein thehand-held device communicates with the intermediate device via awireless signal transmission.
 17. The system as recited in claim 1,wherein the programming for transmitting a signal having data indicativeof the gesture based input comprises a remote control user interfaceapplication resident on the hand-held device.
 18. The system as recitedin claim 17, wherein the hand-held device comprises a touch-screendisplay and the remote control user interface application causes displayof one or more screens displaying icons representative of functionaloperations of the target controllable device which are to be interactedwith to provide the gesture based input.
 19. The system as recited inclaim 18, wherein the remote control user interface application isconfigurable to provide one or more screens displaying iconsrepresentative of functional operations of a plurality of controllabledevices including the target controllable device.
 20. The system asrecited in claim 1, wherein the operating function of the targetcontrollable device that is associated with the gesture based input isuser configurable.
 21. The system as recited in claim 1, wherein thetarget controllable device that is associated with the gesture basedinput is user configurable.